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consistent  error  or  wrong  operation  can  be  diagnosed.  The  system  of 
"error  vectors",  which  is  equivalent  in  power  to  the  "process  network", 
enables  us  to  determine  error  types  committed  by  a  student  in  a  con¬ 
ventional  test,  as  well  as  to  develop  an  adaptive  test  for  diagnosing 
the  misconceptions  possessed  by  a  student.  If  this  procedure  were 
applied  to  adaptive  achievement  testing,  the  validity  of  scoring  would 
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This  study  is  an  attempt  to  improve  the  quality  of  computerized 
adaptive  testing  as  an  integral  part  of  instruction.  An  adaptive 
achievement  test  for  teaching  signed-numbers  operations  was  implemented 
along  with  a  computerized  routing  system  on  the  PLATO  system  in  1977. 

A  close  investigation  of  the  students'  performance-scores  on  the  post¬ 
test  of  this  program  led  us  to  believe  that  a  deeper  level  of  consider¬ 
ations,  not  just  the  data  scored  right  or  wrong,  in  measuring  students 
performance  would  be  needed  in  future  computerized  tests.  Diagnosing 
the  misconceptions  possessed  by  students  is  important  not  only  for 
increasing  efficiency  of  learning  activities  but  also  to  score  test-items 
properly.  Some  problems  could  be  got  right  by  a  wrong  rule  of  operation. 
These  may  be  called  "false  correct s."  It  is  shown  in  this  work  that  in 
some  cases  as  many  as  27  of  the  64  test  items  need  to  be  adjusted  from 
right  to  wrong  if  we  are  to  discredit  "false  corrects".  Finding  all 
types  of  wrong  rules  of  operation  associated  with  a  particular  teaching 
method  for  integer  operations  was  tried  by  performing  error  analysis  on 
some  paper-and-penc i 1  as  well  as  on-line  conventional  tests. 

Our  approach  to  performing  error  analysis  on  the  conventional 
tests  is  to  generate  "error  vectors"  from  item-responses,  instead  of 
generating  responses  from  a  "process  network"  as  some  researchers  have 
done.  By  checking  error  vectors  generated  from  a  particular  set  of 
items,  a  consistent  error  or  wrong  operation  can  be  diagnosed.  The 
system  of  "error  vectors",  which  is  equivalent  in  power  to  the  "process 
network",  enables  us  to  determine  error  types  committed  by  a  student  in 


I 


•MMMti 


'f 


a  conventional  test,  as  well  as  to  develop  an  adaptive  test  for  diagnosing 
the  misconceptions  possesed  by  a  student.  If  this  procedure  were  applied 
to  adaptive  achievement  testing,  the  validity  of  scoring  would  be  greatly 
improved. 
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INTRODUCTION 

Birenbaum  &  Tatsuoka  (1980}  have  identified  various  erroneous 
rules  of  operations  in  signed-number  arithmetic.  Many  researchers 
have  investigated  errors  observed  in  classroom  teaching  or  performance 
data  from  achievement  tests  administered  in  conjunction  with  instruction. 
Error  analysis  had  mainly  been  utilized  in  teaching  and  instructional 
areas  until  Brown  6  Burton  (1978)  introduced  "BUGGY",  an  adaptive 
testing  system  by  which  students'  misconceptions  can  be  diagnosed.  The 
computer  algorithm  of  BUGGY  is  based  on  a  (universal)  cognitive  process 
network  of  addition  and  subtraction  problems  of  whole  numbers.  In 
order  to  construct  such  a  network,  error  analysis  of  the  subject  area 
must  be  thoroughly  conducted  first  so  that  the  mechanics  of  the  mental 
processing  in  achieving  a  given  task  in  the  area  becomes  clear.  Theories 
of  cognitive  processes  have  an  important  role  in  determining  and 
constructing  a  process  network  of  problem  solving  tasks. 


2 

Our  approach  described  in  this  study  is  somewhat  different  from 
the  BUGGY  approach.  Constructing  a  universal  process  network  of  any 
given  task,  in  our  case  the  computational  skills  of  signed-number  oper¬ 
ations,  is  very  difficult.  Moreover,  our  experimental  study  investigating 
the  effect  of  different  instructional  backgrounds  on  error  types  indicates 
that  there  are  significant  differences  in  mental  processing  to  arrive 
at  the  answers  among  students  who  studied  different  instructions.  Thus, 
the  outcome  of  error  analysis  resulted  in  discovering  quite  a  difference 
in  the  types  of  errors  committed  by  each  group.  This  preliminary 
result  of  the  experiment  suggests  that  there  might  not  be  a  unique 
universal  process  network  which  is  applicable  to  all  error  types  and 
attributable  as  the  source  of  each  error.  Errors  depend  on  different 
instruct ional  methods  in  which  the  conceptual  framewrok  of  the 
problem  is  treated  by  different  approaches.  Considering  the  above- 
mentioned  difficulties,  a  method  of  diagnosing  misconceptions  which  is 
more  adjustable  to  a  change  of  instructional  method  is  introduced. 

Our  approach  of  using  "error  vectors"  generates  a  binary  vector  of 
erroneous  and  correct  actions  taken  by  the  student.  Elements  of  an 
error  vector  comprise  all  possible  operations  in  doing  a  problem.  Thus, 
for  an  addition  problem  involving  signed  numbers  (positive  and  negative, 
integers,),  a  three-dimensional  binary  vector  represents  the  actions 
taken  (=1)  or  not  taken  (=0)  in  getting  the  absolute  value  of  the  answer. 
The  successive  elements  represent  the  following  actions:  (1)  Doing 
the  correct  operation;  (2)  adding  the  absolute  values  of  two  numbers; 

(3)  subtracting  them.  Separately,  a  six-dimensional  binary  vector 
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does  the  same  for  the  action  taken  or  not  taken  in  getting  the  sign  of 
the  answer.  The  matrix  product  of  these  two  vectors  yields  a  set 
of  18  "events",  each  being  a  combination  of  two  possible  errors 
associated  with  the  operations  of  taking  absolute  values  and  signs  in 
the  answer. 

When  the  two  vectors  for  each  of  several  problems  have  been  generated 
from  each  student's  responses,  the  elementwise  multiplication  of  all  the 
vectors  of  each  type,  the  "absolute  value"  vector  and  the  "sign"  vector, 
respectively,  produces  a  pair  of  error  vectors  for  the  entire  set  of 
problems.  If  the  student  behaved  consistently,  applying  his/her  rule 
throughout  the  problems,  then  the  final  error  vectors  will  contain  just 
one  1  and  all  other  elements  will  become  zeros.  Therefore  the  matrix 
product  of  the  two  dimensions,  representing  the  operations  of  absolute 
values  and  signs  to  the  answer  will  uniquely  determine  the  type  of 
misconseption,  if  any. 

The  errors  diagnosed  by  the  vector  approach  were  compared  with  the 
results  obtained  from  two  different  approaches:  One  is  examining  a 
response  pattern  of  the  answers  to  the  problems  as  a  whole  and  deducing 
the  student's  rule  of  operations  intuitively  (Bi renbaum  &  Tatsuoka,  1980; 
Neches,  1978);  the  other  is  interviewing  a  few  students  and  asking 
them  to  tell  us  their  rules  of  operation  and  reasoning.  It  was  confirmed 
that  the  error  vector  system  works  equally  well  as  the  other  approaches. 

In  this  paper,  the  error  vector  system  for  addition  problems, 
which  is  based  on  the  instruction  given  by  the  teachers  at  Urbana  Junior 
High  School  in  their  classes,  will  be  introduced  and  then  the 
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error  vector  system  for  subtraction  problems  will  be  introduced  and 
discussed  briefly. 

An  adaptive  test  capab'e  of  diagnosing  the  student's  misconceptions 
in  the  addition  problems  was  programmed  on  the  PLATO  system  (a 
Computer-based  Education  System  at  the  Un i vers i ty  of  Illinois)  and 
it  was  tried  with  180  seventh  graders  in  January,  1980.  Evaluation  of 
the  results  indicated  an  interesting  change  in  the  student's  mental 
processing  activity  that  has  probably  never  been  observed  in  the 
traditional  scoring  procedure  of  any  tests.  Moreover,  it  was  observed 
that  quite  a  number  of  students  switched  the  rules  of  operations  from 
one  to  another  --  sometimes  from  the  right  rule  to  a  wrong  rule, 
sometimes  the  other  way  around.  By  observing  these  shifts  among  different 
error  types,  one  can  conclude  that  there  are  certain  errors  which  can 
eventually  be  converted  to  the  right  rule  of  operation  without  having 
to  repeat  the  same  instruction  after  the  test,  but  some  errors  will  never 
behave  like  these  innocuous  errors.  This  study  suggests  that  it  is 
possible  to  develop  a  quantitative  model  of  scoring  so  that  each  error 
type  will  be  assigned  a  real  number  which  is  an  indicator  of  the 
seriousness  of  the  error.  Some  serious  errors  are  due  to  musunder- 
standing  of  an  important  concept  while  others  are  based  on  a  simple 
mistake  which  can  be  corrected  very  easily. 
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ERROR  VECTORS  FOR  ADDITION  PROBLEMS 
OF  SIGNED-NUMBER  COMPUTATIONS 

Our  approach  to  diagnosing  erroneous  rules  of  signed-number 
operations  is  to  generate  a  binary  vector  from  a  student's  response  by 
setting  each  element  of  the  vector  to  1  or  0  according  to  correct  and 
erroneous  actions  taken  by  the  student.  The  elements  of  the  "error 
vector"  comprise  all  possible  operations  in  doing  the  problem.  Deter¬ 
mining  the  elements  of  the  vector  requires  a  careful  examination  of  the 
logical  flow  of  steps  involved  in  reaching  the  answer  to  a  given 
problem.  Although  it  Is  not  necessary  to  have  a  procedural  network  of 
the  computational  operations  of  signed-number  arithmetic,  all  possible 
errors  that  might  occur  in  students'  reponses  must  be  considered  as 
the  dimensions  of  the  error  vector.  Selection  of  the  dimensions  is  crucial 
in  order  to  increase  the  capability  of  diagnosing  misconceptions  possessed 
by  the  student  by  the  error  vector  system. 

Instruction  and  error  vectors:  A  bA-item  free  response  test  composed 
of  addition  and  subtraction  problems  of  signed-number  computation 
(Appendix  1)  was  administered  to  127  eighth  graders  of  a  junior  high 
school,  upon  completion  of  three  weeks  of  instruction.  The  instructional 
method  and  the  students'  performance  over  the  three-week  period  were 
carefully  observed  and  task  analysis  of  the  teachers'  instruction  was 
carried  out.  A  logical  flow  chart  of  the  task  components  was  summarized 
in  Figure  1. 

Insert  Figure  1  about  here 
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A  main  emphasis  of  the  teacher's  instructional  method  was  to  identify 
the  problem  task  first  and  apply  an  appropriate  rule  of  operation  to 
the  problem.  The  rules  of  operation  consist  of  two  different  types 
of  task  for  addition  problems  and  one  extra,  essential  type  for 
subtraction  problems. 

In  the  addition  problems,  the  student  must  identify  whether  or  not 
the  two  numbers  have  the  same  sign.  If  the  signs  of  the  two  numbers 
are  the  same,  then  they  add  the  absolute  values  of  the  two  numbers 
and  take  the  common  sign  of  the  two  numbers  for  the  answer.  If  the 
sign  of  the  two  numbers  are  different,  then  they  subtract  the  absolute 
value  of  the  smaller  number  from  the  absolute  value  of  the  larger 
number  and  take  the  sign  of  the  larger  number.  As  can  be  seen  in  Figure  1 
the  first  decision  to  be  made  is  which  operation  —  adding  the  two  abso¬ 
lute  values  or  subtracting  one  from  the  other  --  i s  to  be  done,  A  few 
possible  errors  in  making  the  decision  can  be  involved.  They  are  listed 
in  the  upper  half  of  Table  1  as  the  category  of  errors  related  to  absolute 
value  operations.  The  first  digit  of  the  error  code  is  uniformly  1, 
signifying  that  this  is  the  first  error  vector.  The  second  digit 
denotes  the  element  number. 

Insert  Table  1  about  here 

The  elements  of  this  error  vector  (referred  to  as  the  absolute-value 
error  vector  hereafter)  represent  the  three  actions  that  may  be  taken, 
and  are  1  or  0  as  follows; 


Table  1 

Elements  of  "Error  Vectors"  in  the  Addition  of 
Signed-Number  Operation  Such  as  a  +  -b 


) 

Category  of  Error  Error  Code  Description  of  Operation 


Related  to 

11 

right  operation 

Absolute  value 

12 

adding  the  absolute  values  of  the 
two  numbers 

Operations 

13 

subtracting  the  absolute  value  of 
the  smaller  number  from  the  absolute 
value  of  the  larger  number 

Determining  the  Sign 

of  Answers 

21 

sign  of  the  larger  number 

22 

sign  of  the  smaller  number 

23 

taking  always  a  +  sign 

24 

taking  always  a  -  sign 

25 

taking  the  sign  of  the  first  number 

taking  the  sign  of  the  second  number 


26 


1=1  if  the  right  operation  is  taken 
L  =  0  i f  not 

/■  -  1  if  the  absolute  values  of  the  two  numbers 
J  are  added 

^  =  0  i f  not 

r  =  i  if  the  absolute  values  of  the  two  numbers 
J  are  subtracted 

^  =  0  i f  not 

The  second  decision  involved  in  the  addition  operation  is  to  choose  a 
right  sign  for  the  answer.  If  the  signs  of  the  two  numbers  are  the  same, 
the  right  sign  for  the  answer  is  the  common  sign  of  the  two  numbers. 

But  if  the  signs  of  the  two  numbers  are  different,  then  the  sign  of  the 
larger  number  must  be  chosen.  The  six  possible  errors  involved  in 
choosing  the  sign  for  the  answer  are  listed  in  the  lower  section  of 
Table  1  (.referred  to  as  sign  error  vector  hereafter).  These  elements  are 
determined  in  the  same  manner  as  the  absol ute-va 1 ue  error  vector. 
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y1 

y2 

y3 

y4 

y5 

y6 


1  if  the  sign  of  the  larger  number  is  taken 
0  if  not 

1  if  the  sign  of  the  smaller  number  is  taken 
0  i f  not 

1  if  +  sign  is  taken 
0  if  not 

1  if-  sign  is  taken 
0  i f  not 

1  if  the  sign  of  the  first  number  is  taken 
0  if  not 

1  if  the  sign  of  the  second  number  is  taken 
0  i f  not 


As  mentioned  earlier,  the  elements  of  the  two  vectors  are  determined 
by  task  analysis  and  the  results  of  an  error  analysis  performed  on  real 
data.  There  are  more  possible  errors  than  those  described  in  Table  1, 
but  the  mechanism  of  diagnosing  misconceptions  possessed  by  the  student 
is  illustrated  by  these  nine  elements  of  the  two  error  vectors  in  order 
to  simplify  a  complex  algorithm  on  which  the  actual  computer  program  of 
automated  error  analysis  is  based.  New  elements  can  be  easily  added  to 
both  the  error  vectors  without  causing  much  change  in  the  computer 
algorithm.  At  the  same  time,  elements  that  are  not  descriptive  of 
actions  taken  by  any  student  can  be  dropped  so  that  the  dimensionality  of 


the  vector  can  be  reduced. 
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Mechanism  of  error  vector  system:  Suppose  a  student  responded  to  three 
items  as  follows:  -5  +  6  =  +11,  +7  +  “9  =  +16  and  -3  +  =  +7.  For  the 

first  item,  -5  +6,  each  element  of  the  3x6  matrix  product  of  the  two 
error  vectors,  the  absolute  value  and  sign  error  vectors,  is  produced 
by  following  the  specifications  of  the  rules  given  in  Table  1,  Table  2 
is  the  summary  of  responses  to  the  item.  Since  the  student's  response  is 
+  11,  the  cells  Cl 2 , 2 1 ) ,  02,23)  and  (12,26)  in  which  +11  is  seen,  represent 
the  algorithms  possibly  used  by  the  student.  The  response,  +11,  to  the 
problem  may  be  obtained  by  adding  the  two  numbers  and  taking  the  sign  of 
the  larger  number,  always  taking  a  +  sign  for  the  answer,  or  taking  the 
sign  of  the  second  number.  This  is  equivalent  to  saying  that  the  matrix 


Table  2 

Responses  to  Item  -5+6  for  18  Events 
(Algorithms),  Binary  Error  Vectors  and  Their  Matrix  Product 
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kJ 


product  of  two  vectors  (010)  and  (101 00 1 J  yields  1  in  the  cells  (2,1), 

(2,3)  and  (2,6).  Leaving  the  three  rules  of  operation  as  the  student's 
possible  algorithm,  we  go  on  to  the  second  item  to  see  if  the  rule  of 
operation  can  be  determined  uniquely. 

For  the  next  item,  +7  +  “9  =  +16,  Table  3  shows  the  responses 
obtained  by  using  the  18  possible  algorithms.  The  cells  (12,22),  (12,23)  and 
(12,25)  contain  the  numbers  equal  to  the  student's  answer,  therefore  his/her 

Table  3 

Responses  to  I  tern  +7  +  -9  for  18  Algorithms, 

Binary  Error  Vectors  and  Their  Matrix  Product 


Error  Code 

21 

22 

23 

2A 

25 

26 

11 

-2 

+2 

+2 

-2 

+2 

-2 

12 

-16 

+  16* 

+16* 

-16 

+16* 

-16 

13 

-2 

+2 

+2 

-2 

+2 

-2 

The  Matrix  Product  of  two  Error  Vectors 
sign  I 


absolute 

value 


0  110  10 
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possible  rules  of  operation  are:  Adding  the  two  numbers  and  taking  the 
sign  of  the  smaller  number,  always  taking  a+  sign  in  the  answer,  or 
taking  the  sign  of  the  first  number.  In  other  words,  the  matrix  product 
of  the  two  vectors,  absolute-value  and  sign  error  vectors,  (010)  and  (.011010) 
predicts  these  three  possible  rules  of  operation  by  yielding  1's  in  the 
cells  corresponding  to  these  algorithms.  If  the  student  behaved  consistently 
for  responding  to  the  items,  -5  +6  and  +7  +  -9,  then  elementwise  multi¬ 
plication  of  the  two  matrix  products  produces  only  one  non-zero  element, 
thus  signalling  the  only  possible  rule,  (12,23)  or  adding  the  two  numbers 
and  always  taking  a  +  sign  for  the  answer.  This  rule  yields  a  response, 

+7  to  the  third  item  -3  +  -4  which  matches  with  the  student's  answer 
for  the  item. 

The  procedures  illustrated  in  the  above  example  may  be  stated  more 
generally  as  follows.  In  order  to  determine  a  consistent  rule  of  operation 
uniquely,  it  is  necessary  to  consider  a  set  of  appropriate  types  of  items. 
After  the  two  binary  vectors  for  each  of  several  problems  have  been 
generated  from  the  student's  responses  to  these  problems,  the  elementwise 
multiplication  of  all  the  vectors  of  each  type,  the  absolute-value  vector 
and  sign  vector,  respectively,  produces  a  pair  of  error  vectors  for  the 
entire  set  of  problems.  If  the  student  behaved  consistently,  applying 
a  single  rule  throughout  the  problems,  there  will  be  only  one  1  in 
each  of  the  two  error-vector  products.  Thus  his/her  consistent  rule  can 
be  determined  uniquely.  If  the  student  does  not  behave  consistently  and 
applies  different  rules  of  operation  for  each  of  the  problems  in  the  set, 
then  diagnosing  his/her  error  will  be  very  difficult  by  any  method. 
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Figure  2:  The  Performance  of  Student  if 2 


Examples  of  Error  Vectors  for  Addition  Problems 
Committed  by  Two  Students  in  the  November  Data 


Task  § 

Student  #1 

Response  Pattern 

Student  #2  Response  Pattern 

6 

6+4=10 

1 

1 

15 

-6+4=-2 

101 

100110* 

1 

010 

100110 

0 

3 

1 2+ — 3=9 

101 

101010 

1 

010 

101010 

0 

5 

-3+12=9 

101 

101001 

1 

010 

101001 

0 

10 

-l4+-5=-19 

110 

110111 

1 

110 

110111 

1 

11 

3+-5=s-2 

101 

100101 

1 

010 

100101 

0 

14 

-5+~7=-12 

110 

110111 

1 

110 

100110 

1 

7 

8-6=2 

e2 

0 

14 

0 

8 

-l6-(-7)=-9 

e2 

0 

-23 

0 

16 

2-1 1=-9 

e2  (e2, 

11,21)  0 

-13  (el, 12,  21)0 

13 

-3-+12=-15 

e2 

0 

-15 

1 

1 

-6-  (-8 ) =2 

eZ 

0 

14 

0 

12 

9-l-7)  =  16 

e2 

0 

16 

1 

4 

i  -  C- 1  o ) = 1 1 

e2 

0 

1 1 

1 

2 

-7-9=-l6 

e2 

0 

-16 

1 

9 

-12-3=-15 

e2 

0 

-15 

1 

*  "absolute  value"  and  "sign"  error  vectors. 

For  Student  #2,  elementwise  multiplication  of  the  two  sets  of  error  vectors  yields: 

right  add  two  subtract  two  right  sign  of  +  sign  of  sign  of 

numbers  numbers  minimum  first  second 

(0  1  0)(1  0  00  0  0) 

The  rule  of  operation  which  student  #2  used  is  that  two  numbers  are  always 
added  and  the  right  signs  (sign  of  the  larger  number)  were  taken. 


e2:  Bring  down  the  first  number  correctly,  change  the  subtraction  sign  to  + 
and  does  not  change  the  sign  of  the  second  number 
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Examples:  One  of  the  most  common  errors  in  signed-number  arithmetic  is 
adding  the  two  numbers  and  taking  the  sign  of  the  larger  number.  Student 
#2  whose  error  vectors  and  response  pattern  are  shown  in  the  right-hand 
segment  of  Table  4,  used  this  erroneous  rule  consistently  for  all  six 
addition  problems,  with  at  least  one  negative  number,  that  appear  in  the 
second  column  of  the  table.  Figure  2  is  a  copy  of  the  display  on  the  PLATO 
system  showing  the  performance  of  student  #2  on  the  64- i tern  signed-number 
test  given  in  Appendix  1.  The  binary  error  vectors  of  the  addition  problems 
shown  in  Table  4  indicate  that  all  six  absolute-value  error  vectors  have 
1  in  the  second  column  while  1  is  in  the  first  column  of  all  sign  error 
vectors.  This  means  Student  §2  committed  the  error  described  earlier  in 
this  example,  and  obtained  the  right  answer  to  Task  10,  -14  +  -5,  coincidentally. 
In  order  for  us  to  determine  this  error  uniquely,  Student  Hi  needs  to  have 
taken  one  of  the  three  pairs  of  Tasks,  (15,5),  (3,11)  and  (5,14).  It  is 
obvious  that  other  combinations  of  tasks  cannot  provide  a  unique  rule  of  the 
student's  misconception  by  the  error-vector  system  approach. 

Efficient  ways  to  choose  the  appropriate  set  of  items  so  as  to  determine 
any  consistent  rules  of  operation  uniquely  by  the  error  vector  system  are 
very  crucial  and  will  be  investigated  in  a  future  study.  This  is  especially 
necessary  when  the  mechanism  of  the  error  vector  system  is  applied  for 
computerized  adaptive  testing.  It  is  best  to  diagnose  an  error  by  giving  the 
minimum  number  of  items. 

Insert  Table  4  about  here 


Insert  Figure  2  about  here 
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Let  us  consider  the  six  different  types  of  addition  problems  and 

denote  them  as  follows: 

A:  -  larger  number  +  smaller  number  =  -L+S 
B:  -  smaller  number  +  larger  number  =  -S+L 
C:  smaller  number  +  -  larger  number  =  S+-L 
E:  larger  number  +  -  smaller  number  =  L+-S 

E:  -  smaller  number  +  -  larger  number  =  -S+-L 
F:  -  larger  number  +  -  smaller  number  =  -L+-S 

The  18  "events",  represented  by  the  cells  of  the  3xb  matrix  product 

of  the  two  binary  error  vectors,  will  uniquely  determine  the  rule  used 

by  the  student  when  we  form  the  elementwise  product  of  such  matrices  for 

an  appropriate  subset  of  the  six  problem  types.  The  above  example  required 

two  problems,  of  the  B  and  C  types,  to  determine  the  student's  rule  uniquely. 

By  examining  all  combinations  of  error  vectors,  both  the  absolute  value 

and  sign  vectors.  The  Tasks  A  through  F  in  addition  operation  have  the 

capability  of  determining  and  diagnosing  consistent  rules  of  operation 

uniquely  so  long  as  the  elements  of  the  two  vectors  are  limited  to  the 

present  specifications  of  nine  types. 

Adaptive  testing:  An  adaptive  test  for  signed-number  addition  problems 

was  programmed  on  the  PLATO  system  and  used  with  about  160  seventh  graders 

at  the  junior  high  school,  who  had  ju.t  completed  computerized  instruction 

on  signed-number  operations,  and  taken  the  C4-item  free  response  linear 

test.  The  adaptive  test  was  designed  to  repeat  a  cycle  of  testing,  diagnosing 

and  reviewing  three  times.  After  the  first  set  of  problems  is  analyzed  by  the 

error  vector  system,  the  test  program  presents  a  statement  telling  the  student 

his/her  erroneous  rule  and  then  giving  him/her  a  brief  review  lesson  of  addition 

problems.  Upon  completion  of  the  review  lesson,  the  second  set  of  problems  is  adminis- 

i 
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tered.  If  the  student  acquires  the  right  rule  (which  the  error  vector  system 
also  judges  from  the  responses  to  the  items),  then  the  testing  will  be  ter¬ 
minated  because  the  student  has  completed  the  program. 

Table  5  is  a  copy  of  the  display  on  the  PLATO  system  after  error  analyses 
for  50  students  had  been  completed.  The  numbers  in  the  parentheses  designate 
types  of  errors  in  each  testing.  Fifteen  out  of  50  students  arrived  at  the 
right  rule  (11,21)  and  seven  students  mastered  taking  the  right  sign  for 
the  answer  but  failed  to  choose  the  right  aboslute  value  operation  con- 
s i stent  1 y. 

The  first  student,  #1,  has  a  pattern  of  (21 ) (21 )  (1 1 ,21 ) .  Interpretation 
of  Code  (21)  with  the  description  given  in  Table  1,  taking  the  sign  of  the 
larger  number  in  the  answer,  makes  clear  what  happened  with  Student  #1. 

He  took  the  right  sign  for  his  answer  but  not  the  absolute  values.  After 
reviewing  the  instruction  twice,  he  learned  how  to  get  the  right  absolute 
value  for  the  answer. 

Student  #2  responded  to  the  problems  either  randomly  or  using  a  rule 
not  determined  by  the  error  vector  system.  But  her  second  try  is  (23);  in 
other  words,  she  always  takes  the  +  sign  in  the  answer.  Diagnosis  of  her 
error  type  and  reviewing  the  instruction  were  not  effective  for  her. 

Student  #5  repeated  (13,21)  twice  --  that  is,  subtracting  two  numbers 
and  taking  the  sign  of  the  larger  number.  But  on  the  third  try,  his  pattern 
changed  to  an  entirely  new  rule.  #10  repeated  the  same  pattern  as  #5  twice, 
but  the  third  pattern  converged  to  the  right  operation. 


Insert  Table  5  about  here 


Table  6  shows  the  summary  result  of  error  analysis  performed  on  the 
64-item  test.  Since  the  linear  test  did  not  give  any  feedback  during  testing 
time,  students  tended  to  hold  their  rule  of  operation  throughout  the 
period  even  if  it  was  wrong. 

Insert  Table  6  about  here 


ERROR  VECTORS  FOR  SUBTRACTION  PROBLEMS 

The  teachers  at  the  junior  high  taught  their  students  to  solve 
subtraction  problems  by  converting  them  to  addition  problems.  The  rule 
is  that,  without  changing  the  first  number,  students  must  change  the  operation 
sign  -  to  +  ,  then  change  the  sign  of  the  second  number.  By  so  r  .ing,  all 
subtraction  problems  will  be  converted  to  addition  problems.  Let  us  denote 
the  subtraction  tasks  and  an  elementary  addition  tas1<  in  the  6b- i tem  test 
as  fol lows : 


P: 

8-6 

K: 

i-  (-10) 

G: 

2-1 1 

L: 

-b-  (-8) 

H: 

-7-9 

M: 

-Id- (-7) 

1  : 

-12-3 

N: 

-3-+12 

J: 

9-(-7) 

X: 

6+4 

There  are  three  kinds  of  mistakes  that  students  make  when  converting 
a  subtraction  task  to  an  addition  problem.  The  first  mistake  is  not  changing 
the  sign  of  the  second  number  (the  code  110  was  assigned  to  this  operation). 


1 

hi 


Table  5 

The  Number  Line  Method 

Error  Analysis  of  Addition  Problems  in  Adaptive  Testing 


Student  #  Error  Type 


Student  tt 


Error  Type 


1 

(21)  (21) (11,21) 

26 

(25)  0  0 

2 

0  (23) (23) 

27 

(13,26) (21) (13,21) 

3 

(24) (24) (24) 

28 

(13,24) (13,24) (25) 

4 

(26)  0 

29 

(11) (21) (11,21) 

5 

(13,21) (13,24) (26) 

30 

(13,23) (13,23) (11, 

6  ' 

(23) (23)  0 

31 

(11,21) 

7 

(23) (11,26)  0 

32 

0  0  (13) 

8 

0  (24)  (11,21) 

33 

0  0  (11,21) 

9 

C13) (13) (13,24) 

34 

0  0  0 

10 

(13,21) (13,21) (11,21) 

35 

(13,23) (23) (13,23) 

11 

0  (21) 

36 

01,21) 

12 

0  (25) 

37 

0  (13) 

13 

0  0  (13,24) 

38 

(21) (21) 

14 

(24) (24) (12,24) 

39 

(21) (13,21)  (21) 

15 

0  (25) (13,24) 

40 

(21) (13,21) (21) 

16 

(13,21) (21) (23) 

41 

(23) (23) (23) 

17 

(11,24) (24) (21) 

42 

0  0  0 

18 

(24) (11,21) 

43 

0  (21)(11,21) 

19 

0  0  (24) 

44 

(13)  0  (21) 

20 

0  (25) (25) 

45 

0  (11,21) 

21 

(12,24)  (12,24)  (21) 

46 

(25) (24) (12,24) 

22 

(13,23) (11,26) (13,26) 

47 

(24) (21) (24) 

23 

(12) (11,21) 

48 

0  (11,24)  (11,21) 

24 

0  (11,21) 

49 

(11,21) 

25 

0  (23)  0 

50 

0  (24) (24) 

Table  6 

Error  Types  Over  Four  Parallel  Subtests  of  the  First 
Test  in  Group  1  (Addition  Problems) 
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Less  Consistent  Errors 

More  Consistent  Errors 

s  # 

Error  Type 

Score  S# 

Error  Type 

Score 

40 

(11,21) (11,21) (11,21) (11,21) 

19 

52 

(21) (11,21) (11,21) (11,21) 

19 

17 

(11,21) (13,21) (21) (11,21) 

18 

79 

(13,21) (11, 21) (11, 21) (11) 

18 

12 

(11) (11) (11 ,21)  0 

16 

45 

(13,21X11,21)  0  (21) 

16 

55 

(11) (11,21)  (21) (11) 

16 

31 

(11) (13) (13, 21) (11, 21) 

15 

44 

(11,21) (24) (21) (13,21) 

15 

73 

(13) (11,21) (11) (11) 

15 

17 

(11,  (13,21) (13) (13) 

14 

20 

(13,21) (21)  0  (13,21) 

14 

70 

(21)  0  (21)  0 

14 

72 

(13,21) (13,21)  0  0 

14 

30 

0  (21)  0  (13,21) 

13 

60 

(11) (13)  0  0 

13 

57 

0  (13)  (13,21) (13) 

12 

18 

(13)  0  (13) (13,23) 

10 

10 

0  0  (13)  0 

9 

23 

(24)  0  (13,24) (24) 

9 

1 

0  (13,24) (13,24) (24) 

8 

7 

(13,24)  000 

8 

8 

(13)  (13) (13) (13) 

8 

54 

(13,24)  0  (21) (25) 

8 

59 

0  0  0  0 

7 

5 

0  (13,23)  0  (13,23) 

6 

67 

(22) (23)  0  0 

6 

9 

(13,24) (24) (12,24) (12,24) 

4 

47 

(22) (13) (23) (13) 

4 

14 

0  0  0  0 

3 

69 

000  (12,23) 

3 

19 

(23) (12,23) (23) (12,23) 

2 

39 

(24) (24) (24) (12,24) 

2 

62 

0  (24) (24) (24) 

2 

71 

0  0  (24)  0 

2 

6 

0  (21)  0  0 

1 

>  w.,- 


3 

(11, 21)  (11, 21) (11, 21) (11, 21) 

20' 

4 

(11, 21)(11,21)(11, 21)(11,21) 

20 

13 

(11,21) (11,21) (11,21) (11,21) 

20 

34 

(11, 21) (11,21) (11,21) (11,21) 

20 

65 

(11,21) (11,21) (11,21) (11,21) 

20 

27 

(11,21) (11) (11,21) (11,21) 

19 

35 

(11) (11,21) (11,21) (11,21) 

19 

36 

(11, 21) (11) (11,21) (11,21) 

19 

41 

(13,21)(11,21)(11, 21)(U,21) 

19 

42 

(13,21) (11,21) (11,21) (11,21) 

19 

58 

(13,21) (11,21) (21)  (11,21) 

18 

37 

(13) (11,21) (11) (11) 

17 

50 

(13,21) (0) (11,21) (11,21) 

17 

29 

(13,21) (13,21) (21) (13) 

14 

53 

(11,21) (11,21)  0  0 

14 

2 

0  (25) (13,25)  (13,25) 

10 

22 

0  (13,24) (13) (13) 

10 

48 

(13, 21) (13, 24) (13, 24) (13, 24) 

10 

46 

0  0  (13,23X23) 

9 

56 

(11) (21) (23)  0 

9 

26 

(25) (13,25) (13,25)  0 

8 

76 

(23) (13, 23) (13, 23) (13, 23) 

8 

16 

0  (26)  0  0 

7 

43 

0  (13) (13)  (13) ) 

7 

24 

(13,26X13,26X13,26X13,26) 

5 

21 

(24) (24) (12,25) (12,25) 

4 

25 

0  0  0  0 

4 

49 

(12) (12) (12,24) (12,24) 

4 

64 

0  (12,23) (12,23)  0 

1 
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The  second  mistake  is  changing  the  signs  of  both  numbers  (referred  to  by 
Code  Oil  nereafter).  The  third  mistake  is  changing  the  sign  of  the  first 
number  instead  of  the  sign  of  the  second  number  (referred  to  by  Code  010 
hereafter).  The  right  conversion  is  denoted  by  Code  111.  Note  that  these 
codes  are  not  error  vectors.  The  converted  tasks  of  the  problem,  10  -  (-1), 
by  using  each  of  the  four  types  are  listed  below: 


Code 

Subtract i on 

Converted  Add i t ion 

110 

10-(-1) 

1 0+  (- 1 ) 

0 

type 

01 1 

1  0-  (-1  ) 

- 1  0+ (+1  ) 

A 

type 

010 

10-  (-1) 

-  1 0+  (-  1 ) 

F 

type 

11  1 

10-1-1) 

10+(+1 ) 

X 

type 

Table  7  is  a  summary  list  of  addition  tasks  (upper  half  of  the  table), 
the  original  subtraction  tasks  along  with  their  converted  addition  tasks. 

As  can  be  seen  easily,  each  problem  of  subtraction  has  four  possible  converted 
tasks  according  to  the  types  of  conversion  111,  110,  011  and  010,  the  first 
being  the  correct  conversion. 

Insert  Table  7  about  here 

Each  of  the  four  converted  tasks  has  a  pair  of  error  vectors,  the  absolute- 
value  and  sign  vector,  generated  from  the  student's  response. 

Suppose  a  student  answers  the  item  7  -  8  =  ?  and  gets  the  answer  -1. 

Then  we  will  have  four  possible  pairs  of  absol ute- va 1 ue  and  sign  error  vectors 
as  shown  in  the  last  two  columns  of  Table  8.  Each  of  these  pairs  represents 
the  operations  that  must  oe  carried  out  on  the  converted  addition  proolen 
of  that  row  in  order  to  produce  the  result  -1. 
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Table  7 

A  List  of  Converted  Tasks  of  Subtraction  to 
Addition  Problems  by  Four  Possible  Ways 
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Table  8 

Four  Possible  Conversions  of  Task  7  -  8  to 
the  Addition  Problems 


Response  Code  Type  of  Task  Absolute  Value  Vector  Sign  Vector 


-1 

111 

7+-8, 

C 

(1  0  1) 

0  0 

0 

1 

0  11 

-1 

110 

7+8, 

X 

(0  0  1) 

(0  0 

0 

1 

0  0) 

-1 

011 

-7+-8, 

E 

(0  0  1) 

(1  1 

0 

1 

1  1) 

-1 

010 

-7+8, 

B 

(1  o  1) 

(0  1 

0 

1 

1  0) 

If  the  student  knows  the  right  rule  of  operation  for  addition  problems, 
then  he/she  should  be  able  to  carry  out  the  converted  task  correctly. 

Since,  of  the  four  pairs  of  error  vectors,  only  the  pair  in  the  first  row 
represents  the  correct  operation  on  the  converted  addition  task,  it  may  be 
concluded  that  this  student  used  the  conversion  shown  in  the  first  row, 
namely,  the  correct  conversion  (ill). 

f-tore  generally,  in  the  example  of  Table  8,  the  student's  rule  of  converting 
subtraction  to  addition  is  the  Code  111  type  _if_  he/she  can  correctly  answer 
the  Task  C  addition  problem  with  the  right  rule.  If,  on  the  other  hand, 
the  student's  rule  of  operation  for  addition  problems  is  "the  right  rule 
for  taking  the  absolute  value  but  taking  the  sign  of  the  first  number  in 
the  answer"  then  the  student's  conversion  rule  of  subtraction  to  addition 
is  the  Code  010  type. 

The  two  examples  in  Table  4  are  the  eighth  graders  who  took  the  64-item 
signed-number  test  after  the  three  weeks  of  instruction  and  practice  in 
signed-number  operations.  Student  #1  uses  the  right  rule  for  addition 


problems  but  converts  subtraction  into  addition  i  ncorrect  I  y .  His  rule  i 
expressed  by  Code  110.  We  designated  his  rule  by  e2  in  the  table  but  it 
means  that  he  committed  the  wrong  conversion  rule,  110  and  applied  the 
right  rule  of  addition  to  the  newly  converted  addition  tasks.  Student 
ttl  converted  the  subtraction  problems  correctly,  but  her  addition  rule  was. 
wrong:  Adding  the  two  absolute  values  and  taking  the  sign  of  the  larger 
number.  Thus,  her  rule  was  coded  (el,  12, 21).  Tables  given  in  APPENDIX 
show  the  binary  error  vectors  for  Student  #1,  and  APPENCICES  3  and  4  are  the 
performance  of  the  test  and  binary  error  vectors  for  Student  §2.  Further 
discussion  of  the  topic  "Error  Vectors  for  Subtraction  Problems"  will  be  given 
in  a  subsequent  technical  report. 


DISCUSSION 

A  brief  description  of  the  method  using  what  we  termed  binary  error 
vectors  for  diagnosing  mi sconcept ions  possessed  by  students  was  given 
and  its  mechanism  was  illustrated  with  several  examples  obtained  from  real 
data.  A  computer  program  capable  of  finding  240  basic  errors  in  signed-number 
computations  --  subtracting  or  adding  two  absolute  values,  taking  a  parti¬ 
cular  sign  for  the  answer  and  converting  subtraction  to  addition  problems  -- 
has  been  written  on  the  PLATO  system  and  used  for  analyzing  the  64-item 
conventional  test  as  well  as  the  adaptive  test  of  addition  problems.  The 
results  of  error  analyses  performed  on  over  two  hundred  students  will  be 
discussed  in  detail  in  other  technical  reports,  because  the  findings  should 
be  of  interest  to  researchers  in  fields  as  diverse  as  psychometrics, 


instructional  design  and  learning  theory,  thus  making  it  desirable  to 
prepare  several  technical  reports  with  different  foci. 

The  actual  computer  algorithm  is  more  complicated  than  it  was  feasible 
to  describe  in  this  paper,  but  the  basic  mechanism  is  the  same  as 
illustrated  in  the  examples.  Although  the  error-vector  system  was  discussed 
in  the  context  of  signed-number  operations,  the  rationale  and  technique  of 
the  method  can  be  easily  expanded  to  other  areas  of  arithmetic,  algebra,  or 
even  natura I  - 1 anguage  syntax.  Error  components  lor  generic  type  of  error 
vectors  such  as  the  absol ute-va 1 ue  error  vector  or  the  sign  error  vector), 
elements  of  each  error  vector  can  be  determined  by  performing  a  careful 
task  analysis  on  the  subject  area  of  interest.  Great  caution  must  be  taken 
in  the  analysis  because  results  of  our  preliminary  data  analyses  showed 
that  different  instructions  yielded  different  kinds  of  erroneous  rules.  This 
implies  that  the  logical  flow  of  task  components  depends  on  the  type  of 
instructional  methods  that  have  adopted  different  conceptual  frameworks 
to  teach  the  same  topic. 

For  example,  Greeno  (1976)  showed  two  different  procedural  representations 
of  the  concept  (or  procedure)  of  adding  two  fractions.  One  is  the  procedure 
of  finding  equivalent  fractions,  using  a  special  representation  of  fractional 
quantities  and  the  other  is  using  a  set-theoret ic  representation  of  fractional 
quantities.  The  two  approaches  use  different  methods  in  conceptualizing 
the  same  task.  Our  question  is  whether  or  not  different  instructional 
approaches  yeild  different  types  of  erroneous  rules.  This  question  was 
answered  in  the  affirmative  by  our  experimental  study. 


The  experiment  that  was  conducted  last  January  --  by  which  the  relation¬ 
ships  between  characteristics  of  achievement  data  and  their  progressive 
changes  in  various  learning  stages  given  different  instructional  methods 
have  been  investigated  --  revealed  that  two  different  instructional  methods, 
using  the  number  line  and  verbal  stories  to  carry  out  computations  of 
signed-number,  yielded  quite  a  few  different  types  of  errors.  The  results 
of  error  analyses  regarding  this  aspect  will  be  discussed  in  a  future  tech¬ 
nical  report. 

This  finding  raises  a  question  as  to  the  extent  to  which  our  present 
error  vector  system  is  valid  for  the  number  line  method,  since  it  was  con¬ 
structed  for  the  verbal  rules  of  operations  and  stories  approach.  Further 
investigation  on  this  matter  is  underway. 

The  adaptive  test  for  addition  problems  was  designed  and  administered 
to  about  150  students  on  the  PLATO  system,  although  evaluation  of  the  new 
testing  prodedure  and  its  effect  on  teaching  were  not  discussed  in  this  report. 
Selecting  the  minimum  number  of  items  from  an  item  pool  to  determine  a 
consistent  rule  of  operation  for  addition  problems  was  not  much  trouble, 
because  there  were  not  so  many  types  of  error.  But  a  diagnostic  adaptive 
test  for  subtraction  problems  is  not  as  easy  to  construct  as  one  for  addition 
problems.  As  mentioned  earlier  in  the  report,  the  number  of  possible  con¬ 
sistent  erroneous  rules  for  subtraction  problems  increases  drastically  to 
nearly  300.  Thus,  some  new  technology  will  be  needed  to  allow  systematic 
and  economical  item  selection  procedure  in  order  to  determine  any  consistent 
rules  of  operation.  "BUGGY"  by  Brown  and  Burton  apparently  has  its  testing 
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procedure  based  on  "the  universal  process  network"  of  whole-number  arithmetic. 

As  our  experimental  study  demonstrated,  different  instructional  methods 
of  teaching  s inged-number  operations  produced  different  types  of  erroneous 
rules;  moreover  the  task  components  of  each  method  for  solving  a  given  pro¬ 
blem  differ  considerably.  Although  Brown  6  Burton  discuss  the  importance  of 
the  procedural  network  to  develop  a  diagnostic  adaptive  test,  there  cannot 
be  a  unique,  universal  procedural  network  for  signed-number  operations. 

The  item  selection  procedure  used  in  our  adaptive  test  for  addition 
problems  did  not  require  a  procedural  network.  Instead,  the  approach  taken 
was  quite  similar  to  our  method  for  determining  the  specific  wrong  rule  of 
operation  Ci f  any)  by  means  of  the  error-vector  system.  When  a  student  gives 
a  certain  answer  (either  correct  or  incorrect)  to  the  first  item  administered, 
each  of  the  two  error  vectors  (for  the  absolute  value  and  the  sign  respectively) 
will  generally  contain  several  l's,  indicating  that  there  are  several  alternative 
rules  that  he/she  may  have  used  in  getting  that  answer.  The  next  item  to  be 
administered  is  chosen  so  as  to  eliminate  one  or  more  of  these  l's  in  the 
elementwise  product  of  each  of  the  two  successive  pairs  of  error  vectors  (the 
absolute-value  error  vectors  of  the  two  items,  and  the  two  sign  vectors, 
respectively).  Similarly,  the  next  item  is  chosen  to  further  eliminate  l's  in 
the  cumulative  elementwise  products,  and  so  on  until  only  one  1  remains  in 
each  of  the  two  cumulative  products.  It  can  be  shown  that,  provided  a  con¬ 
sistent  rule  of  operation  is  always  used,  at  most  four  items  will  suffice 
to  uniquely  determine  the  rule.  Once  the  rule  is  determined  and  if  it  is 
found  to  be  erroneous,  a  diagnostic  statement  is  made  to  the  student,  and 
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he/she  is  given  a  review  sequence  as  described  on  p.  17.  If  there  is  a  switch 
of  rules  from  item  to  item,  the  errors  are  judged  to  be  "random11,  and  no  diagnosis 
is  made.  This  outcome  is  indicated  by  a  0  in  Tables  5  and  6. 

A  philosophical  cons iderat ion  is  relevant  at  this  point.  Namely, 
is  it  really  meaningful  to  use  the  same  adaptive  test  for  the  dual  purpose 
of  measuring  a  student's  achievement  level  on  the  one  hand  and  diagnosing  his/her 
misconceptions  on  the  other?  In  our  case,  we  treated  addition  and  subtraction 
problems  separately,  and  administered  an  adaptive  test  only  in  addition  in  the 
study  described  in  this  report.  The  reasons  for  this  were  twofold:  Firstly, 
the  wrong  rules  that  were  used  consistently  for  addition  and  those  occurring 
in  subtraction  were  considerably  different;  secondly,  the  values  of  the  ICC 
parameter  were  substantially  different  between  addition  and  subtraction 
problems.  Our  intention  was  to  use  this  adaptive  test  in  addition  for  the  dual 
purpose  mentioned  above.  However,  as  can  be  seen  from  Table  5,  most  students 
were  still  committing  consistent  errors  at  the  time  the  test  was  administered, 
and  hence  it  was  not  very  effective  for  measuring  their  achievement  level.  Only 
when  the  students  approach  mastery,  resulting  in  a  reduction  of  the  incidence 
of  consistent  errors  and  a  relative  increase  of  random  errors  due  to  typing 
mistakes,  momentary  lapses  of  attention  and  the  like,  does  it  seem  meaningful 
to  measure  their  achievement  level.  An  exception  to  this  dictum  may  occur 
when  it  is  desired  to  investigate  how  consistent  errors  change  as  learning 
proceeds.  An  experiment  is  underway-  and  results  of  data  analyses  that  may 
throw  some  light  on  this  question  will  be  discussed  in  a  subsequent  Technical 
Report. 
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Appendix  1 

The  Signed-Number  Test 


I 

II 

Test  Items 

III 

IV 

1. 

-6- (-8) -2 

J  7. 

—1—  C— 10 ) =9 

33. 

—3—  <— 5 ) =2 

49. 

—2  —  ( —  1 1 )=9 

2. 

-7-9—16 

18. 

-2-11—13 

34. 

-4-6—10 

50. 

-5-14—19 

3. 

124-3=9 

19. 

74-5=2 

35. 

154-6=9 

51. 

44-2=2 

4. 

l-C-10 )=1 1 

20. 

3- (-1 2 )=15 

36. 

5-(-7)=12 

52. 

6-(-8)=14 

5. 

-3412=9 

21. 

-1410=9 

37. 

-4413=9 

53. 

-2411=9 

6. 

644=10 

22. 

1048=18 

38. 

2411=13 

54. 

4413=17 

7. 

8-6=2 

23. 

7-5=2 

39. 

4-2=2 

55. 

9-7=2 

8- 

-16-  (-7)— 9 

24. 

-12-(-10)— 2 

40. 

-1 l-(-2)=-9 

56. 

—7— (—5)=— 2 

9. 

-12-3—15 

25. 

-6-4—10 

41. 

-13-4—17 

57. 

-9-7—16 

10. 

-144-5—19 

26. 

-104-1  —  11 

42. 

-74-5—12 

58. 

-104-8—18 

11. 

34-5—2 

27. 

24-11—9 

43. 

64-8—2 

59. 

14-10—9 

12. 

9-(-7)=16 

28. 

6-(-4)=10 

44. 

10-(-l)=ll 

60. 

13-(-4)=l 7 

13. 

-3-412—15 

29. 

-2-411—13 

45. 

-7-49—16 

61. 

1 

1 

ON 

n 

i 

►— * 
o 

14. 

-54-7—12 

30. 

t 

00 

II 

1 

>—* 

46. 

-24-11—13 

62. 

-34-12—15 

15. 

-644—2 

31. 

-543—2 

47. 

-442= -2 

63. 

-846—2 

16. 

2-11—9 

32. 

5-14—9 

48. 

7-16—9 

64. 

4-13—9 
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